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ABSTRACT 

We propose a theory of eigenvalues, eigenvectors, singular values, 
and singular vectors for tensors based on a constrained variational 
approach much like the Rayleigh quotient for symmetric matrix 
eigenvalues. These notions are particularly useful in generalizing 
certain areas where the spectral theory of matrices has tradition- 
ally played an important role. For illustration, we will discuss a 
multilinear generalization of the Perron-Frobenius theorem. 

1. INTRODUCTION 

It is well known that the eigenvalues and eigenvectors of a sym- 
metric matrix A are the critical values and critical points of its 
Rayleigh quotient, x T Ax/|]x|| 2 , or equivalently, the critical val- 
ues and points of the quadratic form x T Ax constrained to vectors 
with unit Z 2 -norm, {x | ||x|| 2 = 1}. If L : R n x R -> R is the 
associated Lagrangian with Lagrange multiplier A, 

L(x,A) = x T Ax-A(||x|| 2 ; -l), 

then the vanishing of VL at a critical point (x c , A c ) £ K" x R 
yields the familiar defining condition for eigenpairs 

A-x. c = A c x c . (1) 

Note that this approach does not work if A is nonsymmetric — the 
critical points of L would in general be different from the solutions 
of Q. 

A little less widely known is an analogous variational approach 
to the singular values and singular vectors of a matrix A £ R mxn , 
with x T Ay /||x|| 2 ||y|| 2 assuming the role of the Rayleigh quotient. 
The associated Lagrangian function L : R m x R n x R — > R is 
now 

L(x,y,a) =x T Ay-a(||x|| 2 ||y||2 - 1). 
L is continuously differentiable for non-zero x, y. The first order 
condition yields 

^4yc/l|yc||2 = ct c x c /||x c || 2 , ,4 T x c /||x c || 2 = o- c y c /||y c || 2 , 

at a critical point (x c ,y c ,a c ) G R m x R n x R. Writing u c = 
x c /||x c || 2 and v c = y c /||y c || 2 , we get the familiar 

Av c = ct c u c , j4 t u c = <t c v c . (2) 

Although it is not immediately clear how the usual definitions 
of eigenvalues and singular values via G} and 

may be gen- 
eralized to tensors of order k > 3 (a matrix is regarded as an 

This work appeared in: Proceedings of the IEEE International Work- 
shop on Computational Advances in Multi-Sensor Adaptive Processing 
(CAMSAP '05), 1 (2005), pp. 129-132. 



order-2 tensor), the constrained variational approach generalizes 
in a straight-forward manner — one simply replaces the bilinear 
functional x T Ay (resp. quadratic form x t j4x) by the multilinear 
functional (resp. homogeneous polynomial) associated with a ten- 
sor (resp. symmetric tensor) of order k. The constrained critical 
values/points then yield a notion of singular values/vectors (resp. 
eigenvalues/vectors) for order-fc tensors. 

An important point of distinction between the order-2 and order- 
k cases is in the choice of norm for the constraints. At first glance, 
it may appear that we should retain the / 2 -norm. However, the crit- 
icality conditions so obtained are no longer scale invariant (ie. the 
property that x c in (3 or (u c ,v c ) in J2| may be replaced by ax c 
or (au c , qv c ) without affecting the validity of the equations). To 
preserve the scale invariance of eigenvectors and singular vectors 
for tensors of order k > 3, the Z 2 -norm must be replaced by the 
Z fc -norm (where k is the order of the tensor), 

||x|| fe = (|a;i| fc H h |a; n | fc ) 1/fe . 

The consideration of eigenvalues and singular values with respect 
to F -norms where p ^ 2 is prompted by recent works fT| |2| of 
Choulakian, who studied such notions for matrices. 

Nevertheless, we shall not insist on having scale invariance. 
Instead, we will define eigenpairs and singular pairs of tensors 
with respect to any 2 p -norm (p > 1) as they can be interesting even 
when p ^ k. For example, when p — 2, our defining equations for 
singular values/vectors {6} become the equations obtained in the 
best rank-1 approximations of tensors studied by Comon |3| and 
de Lathauwer et. al. |4|. For the special case of symmetric tensors, 
our equations for eigenvalues/vectors for p = 2 and p = k define 
respectively, the Z-eigenvalues/vectors and H-eigenvalues/vectors 
in the soon-to-appear paper 1 5 1 of Qi. For simplicity, we will re- 
strict our study to integer-valued p in this paper. 
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2. TENSORS AND MULTILINEAR FUNCTIONALS 

A fc-array of real numbers representing an order-fc tensor will be 
denoted by A = Ki -.jJ G R* x -x<*k. Just as an order-2 
tensor (ie. matrix) may be multiplied on the left and right by a 
pair of matrices (of consistent dimensions), an order-fc tensor may 
be 'multiplied on k sides' by k matrices. The covariant multi- 
linear matrix multiplication of A by matrices Mi = [mH,] £ 



l dlXB1 ,...,M k = [mf h \ k ] eR dfcXSfc is defined by 



A(Mi,...,M k ) ■■= 
z — 'ji=i * — 'jfe =1 



This operation arises from the way a multilinear functional trans- 
forms under compositions with linear maps. In particular, the mul- 
tilinear functional associated with a tensor A £ R dlX '" xdfc and 
its gradient may be succinctly expressed via covariant multilinear 
multiplication: 



\ — \wi v — ^ a k 

A(xi,...,x. k )=2__,. ••2 J aii-JH~n 

*■ — 'ji=i *■ — 'js,=i J1 ■'* 

V x ,4(xi, . . .,Xfc) = A(xi, . . . ,Xi_i,7 di ,x i+ i, . . . ,x fe 



Note that we have slightly abused notations by using A to denote 
both the tensor and its associated multilinear functional. 

l" x "' x ° is called symmetric 



An order-fc tensor [a^...^] £ 



if a 



J<r(l)-J CT (fc) 



aj 1 ,..j k for any permutation a 6 Bfc. The 
homogeneous polynomial associated with a symmetric tensor A = 
and its gradient can again be conveniently expressed as 



r — ^ n r — ^ n 

A(x,...,x) =2^. • -2- ! a ^' 
' — 'ji=i z — 'jfc=i 

VA(x, ...,x) =kA(I n ,x,...,x). 
Observe that for a symmetric tensor A, 



■Xj k , (4) 



3. SINGULAR VALUES AND SINGULAR VECTORS 

Let A 6 R d i x ' xd fc. Then A defines a multilinear functional 
A : R dl x ■ ■ ■ x R dfc — » R via 0}. Let us equip R di with the l Pi - 
norm, ||-|| Pi , % = 1, . . . , fc. We will define the singular values and 
singular vectors of A as the critical values and critical points of 
A(xi, . . . , Xfe)/||xi || P1 ■ ■ ■ ||xfc|| Pfc , suitably normalized. Taking 
a constrained variational approach, we let L : R dl x ■ ■ ■ x R dfc x 
R -» Rbe 



L(xi, . . . ,x fc ,cr) := 

A(xi,...,x fc )-o-(||xi|| pl ■■■||x fe || Pfc -1). 

L is continuously differentiable when x* ^ 0, i = 1, . . . , k. The 
vanishing of the gradient, 

VL = (V X1 L, . . . , V Xfc L, V CT L) = (0, . . . , 0, 0) 



gives 



A(J dl ,X2,X3, . . . ,x fc ) = aip pi -i(x.i) 
A(xi,7 d2 ,x 3 , ... ,x fc ) = o-^ P2 _i(x 2 ) 



(6) 



J 4(X1,X 2 , . . . ,3C k -l,Id k ) = tfVpfc-l( x k)) 

at a critical point (xi, . . . , Xfe, a). As in the derivation of J2j, one 
gets also the unit norm condition 



A(J n ,x,x, . . . ,x) = A(x, 7 n ,x, . . . ,x) = 

••• = ^.(x.x, ...,x,J„). (5) 

The preceding discussion is entirely algebraic but we will now 
introduce norms on the respective spaces. Let be a norm 

on R di , i — 1, . . . , k. Then the norm (cf. |6]) of the multilinear 
functional A : R dl x ■ ■ ■ xR dfc -> R induced by | 
is defined as 



Xfc 



|ai , • ■ • , || ||a fc 



\A\U 



sup ■ 



\A(X! 



,Xfc) 



x fc 



where the supremum is taken over all non-zero x; £ 1 ', i = 
1, . . . , k. We will be interested in the case where the 's are l p - 
norms. Recall that for 1 < p < oo, the Z p -norm is a continuously 
differentiable function on R n \{0}. For x = [x%, . . . , x n ] T G R n , 
we will write 

X • [^1; • • • ) «^n] 
(ie. taking pth power coordinatewise) and 

tp p (x) :=[sgn(zi)a5?, . . . , sgn(a;„)a;£] T 

where 

{+1 ifx>0, 
if x = 0, 
-1 ifa;<0. 

Observe that if p is even, then <£ p (x) = x p . The gradient of the 
2 p -norm is given by 



vilxl! 



Vp-i(x) 



The unit vector x, and a in {6j, will be called the mode-i singular 
vector, i = 1, . . . , k, and singular value of A respectively. Note 
that the mode-i singular vectors are simply the order-fc equivalent 
of left- and right- singular vectors for order 2 (a matrix has two 
'sides' or modes while an order-fc tensor has fc). 

We will use the name / pl '"' Pfc -singular values/vectors if we 
wish to emphasize the dependence of these notions on ||-|| Pi , i = 
1, . . . , fc. If pi = ■ • • — pk = p, then we will use the shorter 
name Z p -singular values/vectors. Two particular choices of p will 
be of interest to us: p = 2 and p = fc — both of which reduce to 
the matrix case when fc = 2 (not so for other choices of p). The 
former yields 

A(Xi, . . . ,X 4 ,/ di ,X i+ i, . . .,Xfc) = (TXi, i = 1, . . . ,fc, 

while the latter yields a homogeneous system of equations that is 
invariant under scaling of (xi, . . . ,Xfc). In fact, when fc is even, 
the P -singular values/vectors are solutions to 



A(xi 



,Xfc) = o-x. 



i = 1, 



.fc. 



hp- 1 



The following results are easy to show. The first proposition 
follows from the definition of norm and the observation that a max- 
imizer in an open set must be critical. The second proposition fol- 
lows from the definition of hyperdeterminant Q; the conditions 
on di are necessary and sufficient for the existence of A. 

Proposition 1. The largest l pi '---' p tc singular value is equal to the 
norm of the multilinear functional associated with A induced by 
the norms ll-IL, , . . . , ll-IL,, ie. 



or simply V||x|| p = x p /||x|| p whenpiseven. 



= \\A\\ pl ,..., Pk 



Proposition 2. Let di, . . . , dk be such that 

di — 1 < (dj — 1) for all i = 1, . . . , k, 

and A denote the hyperdeterminant in R dl x ' " xdk . Then is an 
I 2 -singular value of A G R d i x ' xd fc if and only if 

A(A) = 0. 

4. EIGENVALUES AND EIGENVECTORS OF 

SYMMETRIC TENSORS 

Let A G R n x "' x " bean order-fc symmetric tensor. Then A defines 
a degree-fc homogeneous polynomial function A : R" — » K via 
0. With a choice of Z p -norm on R n , we may consider the mul- 
tilinear Rayleigh quotient A(x, . . . , x)/||x||p. The Lagrangian 

L :R n xl^I, 

L(x,A) := A(x,...,x)-A(||x||*-1), 
is continuously differentiable when x =^ and 
VL = (V X L,V A L) = (0,0) 

gives 

A(J n ,x,...,x) = Av?p-i(x) (7) 
at a critical point (x, A) where ||x|j p = 1. The unit vector x and 
scalar A will be called an F -eigenvector and l p -eigenvalue of A 
respectively. Note that the LHS in Q satisfies the symmetry in i|5j. 

As in the case of singular values/vectors, the instances where 
p = 2 and p — k are of particular interest. The Z 2 -eigenpairs are 
characterized by 

A(J n ,x, . . . ,x) = Ax 

where ||x||2 = 1. When the order k is even, the Z ft -eigenpairs are 
characterized by 

A(/ n ,x,...,x) = Ax*" 1 (8) 

and in this case the unit-norm constraint is superfluous since J8j 
is a homogeneous system and x may be scaled by any non-zero 
scalar a. 

We shall refer the reader to |5J for some interesting results on 
^-eigenvalues and l k -eigenvalues for symmetric tensors — many 
of which mirrors familiar properties of matrix eigenvalues. 

5. EIGENVALUES AND EIGENVECTORS OF 

NONSYMMETRIC TENSORS 

We know that one cannot use the variational approach to charac- 
terize eigenvalues/vectors of nonsymmetric matrices. So for an 
nonsymmetric tensor A G R n x " x ", we will instead define eigen- 
values/vectors by — an approach that is consistent with the ma- 
trix case. As J5J no longer holds, we now have k different forms 
of 0: 

A(/„,Xi,xi, . . . ,xi) = ^i<£ p _i(xi), 

A(Xl, J„,X 2 , . . . ,X 2 ) = ^2<£p-l(x 2 ), 

(9) 

A(x fe ,Xfe, . . . ,x fc , I n ) = Pfc^p-i(x fc ). 
We will call the unit vector Xj a mode-i eigenvector of A corre- 
sponding to the mode-i eigenvalue pi, i = 1, . . . , k. Note that 
these are nothing more than the order-fc equivalent of left and right 
eigenvectors. 



6. APPLICATIONS 

Several distinct generalizations of singular values/vectors and eigen- 
values/vectors from matrices to higher-order tensors have been 
proposed in (5| [5] |4] [5] |5J. As one can expect, there is no one 
single generalization that preserves all properties of matrix singu- 
lar values/vectors or matrix eigenvalues/vectors. In the lack of a 
canonical generalization, the validity of a multilinear generaliza- 
tion of a bilinear concept is often measured by the extent to which 
it may be applied to obtain interesting or useful results. 

The proposed notions of I 2 - and l k -singular/eigenvalues arise 
naturally in the context of several different applications. We have 
mentioned the relation between the Z 2 -singular values/vectors and 
the best rank-1 approximation of a tensor under the Frobenius 
norm obtained in (3]I3). Another example is the appearance of l 2 - 
eigenvalues/vectors of symmetric tensors in the approximate solu- 
tions of constraint satisfaction problems 1 10 1 1 ]. A third example 
is the use of l k -eigenvalues for order-fc symmetric tensors (fc even) 
for characterizing the positive definiteness of homogeneous poly- 
nomial forms — a problem that is important in automatic control 
and array signal processing (see |5| |12I and the references cited 
therein). 

Here we will give an application of l k -eigenvalues and eigen- 
vectors of a nonsymmetric tensor of order fc. We will show that 
a multilinear generalization of the Perron-Frobenius theorem 1 13 
may be deduced from the notion of i fe -eigenvalues/vectors as de- 
fined by 0. 

Let A = [ajvij £ R" x '" xn . We write A > if all 
a ii-3h > (likewise for A > 0). We write A>_BifA-B>0 
(likewise for A > B). 

An order-fc tensor A is reducible if there exists a permutation 
a 6 & n such that the permuted tensor 

has the property that for some m £ {1, . . . , n — 1}, bi 1 ..-i k = 
for all ii 6 {1, . . . , n — m} and alli2, . . . , ik G {1, • ■ • , m}. 

If we allow a few analogous matrix terminologies, then A is 
reducible if there exists a permutation matrix P so that 

B = A(P,...,P) G R"x-x" 

can be partitioned into 2" subblocks and regarded as a 2 x ■ ■ ■ x 2 
block-tensor with 'square diagonal blocks' Soo-o £ R mx " xm , 
G R ( "- m)x " x <n_m) , and a zero 'corner block' S 10 ... G 
R (n ~ m)xmx "' xm which we may assume without loss of general- 
ity to be in the (1,0,..., 0)-'corner'. 

We say that A is irreducible if it is not reducible. In particular, 
if A > 0, then it is irreducible. 

Theorem 1. Let A = [a^....,-^ G K nx " ' xn be irreducible and 
A > 0. Then A has a positive real l k -eigenvalue with an l k - 
eigenvector x» that may be chosen to have all entries non-negative. 
In fact, x» is unique and has all entries positive. 

Proof. Let §" := {x G R n | x > 0, |[x|| fe = 1}. For any 
x G §™ , we define 

p(x) := inf{/i G R+ | A(l,x, . . . ,x) < {ix.^ 1 }. 

Note that for x > 0, y3fc_i(x) = x fe_1 . Since S+ is compact, 
there exists some x, G S" such that 

/i(x») = mf{/tt(x) I x G Sfc} =: jtt*. 



Clearly, 

A(I n ,x.,...,x.) <m*x» _1 . (10) 
We claim that x» is a (mode-1) l k -eigenvector of A, ie. 

A(I n , x«, . . . ,x») = ^.x* -1 . 

Suppose not. Then at least one of the relations in 1 101 must hold 
with strict inequality. However, not all the relations in 1101 can 
hold with strictly inequality since otherwise 

A(I n ,x,, . . . ,x„) < ^i.xjj -1 (11) 

would contradict the definition of p, as an infimum. Without loss 
of generality, we may assume that the first m relations in 1101 are 
the ones that hold with strict inequality and the remaining n — m 
relations are the ones that hold with equality. We will write x* = 
[x , xi] T with x G R m , xi G R n ~ m . By assumption, A may be 
partitioned into blocks so that 

A(,Q...o(I m , xo, • . • , xo, xo)+ 

Aoo...\(I m ,-Ko, ...,X ,Xl) H h 

A l...l(/m,Xi, . . . ,Xi,Xi) < ^»Xo~\ (12) 

A\0- ■ -0 (In-m, Xo, . . . , Xo, Xo) + 

A 0--l(/n-m,Xo, . . . ,X ,Xi) H h 

An...i(7 n _ m ,xi, . . . ,xi,xi) = fi,Xi _1 . (13) 

Note that xo / since the LHS of J 1 21 is non-negative. We will 
fix xi and consider the following (vector-valued) functions of y: 

F(y) :=Aoo...o(im,y,...,y>y)+ 

A o--i{Im,y, . . . ,y,xi) h h 

A i...i(J m ,xi, . . . ,xi,xi) - /U»y 

G(y) := A 10 ... (In-m,y, ■ ■ ■ ,y,y)+ 

y,xi) + --- + 

Au...i(7 n _ m ,xi, . . . ,xi,xi) - ^.x^ 1 . 

Let fi, ■ ■ ■ , fm and gi, . . . , g n -m be the component functions of 
F and G respectively, ie. fi's and gfs are real-valued functions 
of y G R m such that F(y) = [/i(y), . . . , ,f m (y)] T and G(y) = 
[fl , i(y),---,3n-m(y)] T . 

By i 1 21 . we get /i(xo) < for i — 1, . . . , m. Since /, is 
continuous, there is a neighborhood B(xo, 6i) C R m such that 
/i(y) < for all y G B(x ,5i). Let S = min{<5i, . . . , S m }. 
Then F(y) < for all y G B(x , 5). 

By 1 1 3i . we get ffj(xo) = for j = 1, . . . ,n — m. Ob- 
serve that if gj is not identically 0, then <7j(y) = 9j(yi, ■ ■ ■ , Vm) 
is a non-constant multivariate polynomial function in the variables 
yi, . . . , y m . Furthermore, all coefficients of this multivariate poly- 
nomial are non-negative since A > 0. It is easy to see that such 
a function must be 'strictly monotone' in the following sense: if 
< y < z and z ^ y, then gj(y) < 3j(z). So for < y < xo 
and y 7^ xo, we get gj(y) < 5j(xo) = 0. Since A is irreducible, 
.i4.io.--o is non-zero and thus some gj is not identically 0. 

Let yo be a point on the line joining to xo within a distance 
S of xo and yo 7^ xo. Then with yo in place of xo, the m strict 
inequalities in 1121 are retained while at least one equality in 1131 
will have become a strict inequality. Note that the homogeneity of 



< 1 21 and 1131 allows us to scale [yo, xi] T to unit ? fe -norm without 
affecting the validity of the inequalities and equalities. Thus we 
have obtained a solution with at least m + 1 relations in i lOi being 
strict inequalities. Repeating the same arguments inductively, we 
can eventually replace all the equalities in 1 1 21 with strict inequal- 
ities, leaving us with <1 It . a contradiction. [We defer the proof of 
uniqueness and positivity o/x* to the full paper.} □ 

A proposal to use the multilinear Perron-Frobenius theorem in 
the ranking of linked objects may be found in 1141 . A symmetric 
version of this result can be used to study hypergraphs 1151 . 
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